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Abstract 

Seniors’writing skills were assessed in 1998 at a medium-sized public university. Blind scoring, a standard 
scoring guide, and trained graders were used. Curricular writing emphasis was assessed through a syllabus 
study, yielding a Curricular Emphasis Score. Controlling for entry-level skill in writing, Writing Score and 
Curricular Emphasis were highly correlated. 

Introduction 

Writing across the curriculum is an emphasis that, like apple pie, enjoys widespread appeal. Ac¬ 
cording to the MLA Commission on Writing and Literature (1985), 47% of 4-year colleges had writing 
across the curriculum programs, and that percentage continues to climb. The belief in its effectiveness 
remains strong in colleges and universities across the land. However, among writing professionals skep¬ 
tics point out that after programs have been set up, assessment is often lax or non-existent. Convincing 
evidence sufficient to satisfy the wary researcher, policy-maker or administrator, controlling for relevant 
variables, is difficult to find. A partial catalogue of these relevant variables would include, at the top of the 
list, pre-existing writing ability as it can be estimated upon college entry. In addition, given the fact that 
increasing numbers of students attend more than one institution on their journey toward graduation, there 
is the factor of the writing emphasis found at multiple institutions. Since the experience gained earlier at 
another institution is beyond the control of a college or university, how does the latter institution deter¬ 
mine what part of the writing proficiency of its graduates was contributed by its courses and what part by 
the prior experience? 

Another major problem facing research comes from the nature of the curriculum. Accreditation 
standards require that virtually all institutions have curricula that foster writing proficiency along with 
other general education skills. Many institutions, including the one that is the object of this presentation, 
address this requirement through both required freshman writing courses and other required courses in 
the curriculum, sometimes designated as having a “writing emphasis.’’The problem that remains is that 
learning does not necessarily follow prescribed patterns laid down in the curriculum. Students may im¬ 
prove their writing in many ways through a myriad of course-taking patterns, including those without any 
official designation as “writing-intensive.” What may be needed is a method that brings to bear an inde¬ 
pendent outside assessment of the writing-intensiveness of the curriculum. Further, such a method must 
allow for individualized measurement of the writing intensiveness for students’ particular courses of study. 

Method 

Such a method was developed by the one of the authors, the late Dr. Edward D. Smith (see Note 
1), and presented at a regional assessment conference. The method requires the collection and review of 
syllabi for all, or almost all, courses in the undergraduate curriculum. Each syllabus is rated by an indepen¬ 
dent rater on a 3-point scale to measure the degree of emphasis on a number of process variables. These 
variables included the following: written communication, oral communication, problem solving, computer 
applications, mathematical applications, international perspectives, and diverse perspectives. The 3-point 
scale defines three degrees: (a) no emphasis that the process variable was being address in the course (score 
= 0); (b) some emphasis that the process variable received some attention at some point (score = 1); and 
(c) strong emphasis that the process variable received emphasis throughout the semester (score = 2). For 
example, strong emphasis in written communication was defined as two or more assigned papers. The 
method has been used at several public institutions in Virginia including the object of the current presen¬ 
tation. In the latter, a sample of syllabi from six departments yielded satisfactory reliability estimated by 
correlations between two independent raters in the approximate range of r=. 7 to .9. 
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Validity was also addressed. At a second institution, validity in the area of written communication 
was indicated in a special study connecting curricular writing intensiveness with seniors’ writing proficiency. 
Writing intensiveness of the curricula in the institution’s divisions (percent of courses with “strong empha¬ 
sis”) correlated significantly with pass-rates on the institution-wide senior writing test; i.e., the measure of 
writing intensiveness in those divisions tracked the proportion of students passing the test, having majors 
in those divisions. This result suggests that, for measuring course process variables, the syllabus method can 
connect such variables with important outcomes. Further, it supports an extension of the study to another 
institution in the particular area of written communication. Another study based in the first institution found 
evidence of validity of syllabus ratings as a measure of writing intensiveness. That study found a significant 
correlation (r = .80 ,p < .01) across departmental programs between syllabus ratings and referrals to the uni¬ 
versity’s Writing Center. 

The question of the stability of the syllabus method also was addressed. The syllabus study was 
repeated for the institution under study during two different years, 1996 and 1999. The same process vari¬ 
ables named above were measured. The primary rater and investigator was the same person, an independent 
outside consultant. For each of the variables, chi squared analysis showed that the combined percent of “some 
emphasis” and “strong emphasis” did not differ significantly for any of the variables between the two years. 
For example, Written Communication showed a mean of 69% in 1996 and 66% in 1999. 

In the current study, the institution is much smaller than the institution that was the focus of the 
previous study reported above. Therefore, students (N = 71) rather than divisions of the institution were the 
focus of study. It thus became necessary to devise an individual measure of the writing intensiveness of the 
individual’s prior course of study. Since this phase of study was quite labor-intensive, a random sample of 25 
students was drawn. The academic records of these students revealed the number of courses taken in each 
discipline. A weighted sum for each student, with weighting by the percent of syllabi that showed “strong 
emphasis” on writing in each discipline from the syllabus study, yielded a Curricular Emphasis Score. 

For the writing outcome measure, since the university did not require a senior writing test it was 
necessary to recruit a sample (N = 71) of the graduating seniors to participate in a senior exit writing test. 

The testing was done in the spring of 1998. Similar writing sample data were collected from samples of 
seniors from 1995 (N = 60) and 1997 (N = 93). Students were selected from the list of prospective graduates, 
using a sampling plan designed to guarantee a good representation of subjects who transferred in freshman 
writing course credits and those who took those courses at the institution. With this one restriction, the 
sample was a random sample. Students were recruited by letter from the provost with a follow-up letter and 
telephone calls from the Director of Assessment. The letter invoked the catalogue graduation requirement to 
participate in various forms of assessment. It also offered a small stipend ($15) for all participants. Testing 
was conducted during the last two weeks of the term. Participation was nearly 90 percent of those students 
eligible and invited to participate. 

The writing sample followed the same procedures and used the same test as was used by fresh¬ 
man writing students for the final of the spring term. For security purposes, it was necessary to schedule 
the testing immediately after the writing prompt was selected and just before finals week when freshmen 
were scheduled to take the examination. Grading also followed the same structured procedures as used for 
freshmen. A standard, structured Scoring Guide was used and all graders, teachers of freshman English, 
were trained for grading consistency. Six dimensions were rated on a 5-point scale. The scale points were as 
follows: 0 = failing, 1 = below average, 2 = average, 3 = above average, and 4 = superior. The dimensions of 
the scoring guide were the following: Summary of Reading, Critique of Reading, Personal Response to the 
Reading, Structure, Correctness, and Style. For this study, a Total Score based on all dimensions was also 
constructed to summarize performance. In addition, instructors assigned a holistic grade on the standard 
4-point scale. Batches of papers were assigned randomly to readers, who were professors of English. These 
assignments were blind with respect to graders’ knowledge of the special status as seniors of those being 
graded. Readers did not read their own students’ papers, and all writers’ identities were disguised. While some 
graders may have suspected, they were not intentionally informed that such a special status existed. Only the 
Director of the Freshman Writing Program, who assisted with the project, was aware of those assignments. 
Superficially, freshmen and senior papers looked identical. 

For statistical purposes, other data were added to the dataset: the mean grades for the two freshman 
English courses, the student’s cumulative GPA at the university, transfer GPA, overall GPA, and the SAT- 
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Verbal score. Due to the importance of examining pre-existing writing ability, the SAT-Verbal score was 
used as a surrogate although it is a fallible measure for that purpose. Support for this choice can be found 
in the results (below) where it will be seen that, of the variables in the study, the SAT-Verbal was most 
highly correlated with the Total Score or composite exit writing score. 

Results 

The syllabus study revealed that approximately two-thirds of courses at the university incorporate 
written communication, defined as the combined ratings of “some” and “strong” emphasis. About one- 
fourth showed a “strong” emphasis or incorporationof two or more writing assignments. For comparison, 
this degree of curricular emphasis was less than the problem-solving emphasis (> 80%), about equal to 
the oral communication emphasis, and greater than all the other dimensions of curricular emphasis. As 
already reported, results were highly consistent over a 3-year period. Senior exit writing results can also 
be compared on the same six dimensions plus the Total Score. These comparisons will not reveal the same 
degree of consistency over time as shown in the syllabus study. 

Comparisons between freshman scores and senior scores were studied in depth in the 1997 study. 
While these were interesting and can be briefly reported, the focus of the current study was on the 1998 
writing study (combined with the 1999 syllabus study), i.e., the relationships between the writing out¬ 
comes variables and the curricular process variables. 

Correlations among all variables were examined with a view toward predicting skill in writing. For 
summary purposes, the Total Score will be used in this proposal in place of the other six writing outcome 
variables, of which it is the composite. Total Score correlated significantly with freshman English grades 
(r = .43), cumulative GPA (r = .38), and SAT-Verbal (r = .57). In addition, freshman English grades corre¬ 
lated (r = .58) with the cumulative GPA. All correlations were significant at the ^><.01 level of confidence. 

The role of transfer was also examined. One comparison was made between participants who 
transferred their freshman English credits from another institution and those who took their freshman 
English course at the university. The dependent variables in the comparison were the holistic grade re¬ 
ceived on the writing test and the Total Score. These differences were not statistically significant. A second 
related comparison looked at native students (students with no transfer credits) versus transfer students on 
the same two dependent variables. Again, these differences were not statistically significant. These results 
were consistent with those found in the 1997 writing study. For 25 students randomly selected to derive a 
Curricular Emphasis Score, the correlatiobetween that score and the Total Score on the writing test was 
r = .40,/> < .05. To control for entry-level skill in writing a partial correlation was conducted on Curricular 
Emphasis and Total Score on the Writing Test, using SAT-Verbal scores as the controlled variable. The 
partial rwas .78,/> < .001. 

Conclusions 

Mean grades in freshman English courses, cumulative GPA, and SAT-Verbal scores all correlate 
significantly with an independent assessment of writing skills of seniors. There are no significant differ¬ 
ences on this senior assessment of writing skills between native and transfer students or between students 
who had taken freshman English at the university or elsewhere. Curricular emphasis on writing correlated 
significantly with this senior assessment of writing skills. This correlation was even stronger after con¬ 
trolling for differences in entry-level writing skills as measured by SAT-Verbal scores. Thus, pre-existing 
writing ability (estimated by the SAT-Verbal) continues to be a strong influence on writing skills later in 
college, continuing through until the time of graduation. However, again there is a highly significant con¬ 
tribution of the curricular emphasis on writing that comes through strongly when pre-existing writing skill 
level is controlled. Certainly, there are flaws in the study; one could wish for a larger sample in the crucial test 
reported here. Nonetheless, the substantial correlationand level of significance are worthy of note. 

Beyond the findings themselves, this study illustrates the use of two methods that have shown 
considerable promise. The first is the syllabus study method, which has been used with good results in at 
least four different institutions. While syllabi present only one window on the important process variables 
related to valued general education outcomes, these early studies are promising. They suggest that instruc¬ 
tor’s educational intents as stated in syllabi are stable and often may be valid indicators of the general 
knowledge and skills that students gain. 
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The second innovative method introduced here is a disciplined process of assessing seniors’writing 
skills. We used trained graders and a structured scoring guide with which graders have become comfortable 
and can use efficiently. We also made the process as blind as we could. The problem of assuring student mo¬ 
tivation to perform well under these institutional circumstances remains difficult. However, compliance with 
the task was good and students’ performance was reasonably in accord with their academic records as regards 
writing. In other circumstances, where real consequences are attached to performance, this approach might 
be even more successful. 

The real message is that of the title of our presentation: “Writing Across the Curriculum Works.” 
We believe our study has overcome at least some of the obstacles that stand in the way of making such a 
claim with confidence, if not certainty. Professors of English as well as many others, who believe in and care 
deeply about fostering writing ability in college, and who have labored toward this end for many years, can 
be encouraged by these results. 


Notes 

1. An earlier version of this paper was presented at the meetings of the Association for the Study of 
Higher Education, Sacramento, CA, November 17,2000. It was also presented at the 14th annual Virginia 
Assessment Group Conference, November 3,2000. The first author particularly acknowledges a debt to 
Edward D. Smith, Ph.D., who was a well-known and respected member of the assessment community in 
Virginia, and a professor of psychology at Longwood University, for many years until his untimely death 
in 2003. Dr. Smith was the inspiration for the method used and its successful application in many contexts 
including the present case, in which he collaborated fully. The first author thanks Mrs. Sherry L. Smith for 
her gracious permission to include her late husband posthumously as honored co-author. 

2. Requests for additional information may be sent to Dennis R. Ridley, Institutional Research and 
Planning, Virginia Wesleyan College, Norfolk, Virginia 23502-5599. 
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